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Background / Context: 

In designing learning environments that support student learning, we face many instructional 
design decisions. These include when and how to provide examples, verbal explanations, 
feedback, and other scaffolding features. In this paper, we investigate instructional guidance as it 
relates to Cognitive Tutor, an intelligent tutoring system that provides students with an 
interactive learning environment. Cognitive tutors for high school mathematics have been 
successful in raising students’ test scores (e.g., Koedinger et ah, 1997; Morgan & Ritter, 2002; 
Pane et ah, 2013) and the Cognitive Tutor Algebra curriculum is being used in over 3,000 
schools nationwide. 

A large number of studies have reported that provision of worked-examples facilitates 
learning (e.g., Atkinson et ah, 2000). For example, Salden et al. (2010) implemented a faded 
worked example (Renkl et al., 2004) to a Geometry Cognitive tutor and reported increased 
learning outcomes. Although there is strong evidence that learning is facilitated by provision of 
worked examples, effects of providing verbal explanations along with examples are mixed at 
best (Lee & Anderson, 2013). However, in our prior study using a laboratory version of the 
algebra tutor (Lee et al., submitted) we found that the effect of providing verbal explanation 
depended on the transparency of the problem structure. Students were able to successfully master 
a problem-solving skill without explicit verbal direction as long as the problem structure was 
apparent enough that they could understand solution steps of the examples. We speculated that 
students in the no-explanation condition could learn to solve problems from non-verbal 
scaffolding methods and constrain their search space from immediate feedback about the 
correctness of their solutions. The immediate feedback helped reduce floundering and discover a 
problem solving rule from a student’s own solutions. Corbett and Anderson (1995), however, 
provided only limited support for the effectiveness of immediate feedback over delayed 
feedback. Using the Lisp tutor, they found immediate feedback led to faster, but not better 
learning than the delayed feedback condition although both conditions led to better learning than 
the no-feedback condition. 

Purpose / Objective / Research Question / Focus of Study: 

This paper reports early results from a large scale experiment evaluating different instructional 
conditions implemented in the Cognitive Tutor Algebra. This study explores the degree to which 
we can teach without verbal explanations and tests whether the results obtained in our laboratory 
will generalize to real classrooms. We developed two versions of the Algebra Tutor which 
differed on whether students were given explanations when they asked for hints and made errors. 
To enable students to learn without explanations, various modifications were made to the tutor, 
guided by three principles that followed from our prior study. The principles were 1) show 
intermediate cognitive steps that are not obvious to students via a worked-out example, 2) 
communicate problem structure by highlighting critical features of the problem, and 3) avoid 
excessive floundering by reducing off-path activity. We tested the effectiveness of the modified 
tutor by comparing to the effectiveness of the standard tutor (i.e., Carnegie Tutor Algebra 2012 
release). 

Population / Participants / Subjects: 

High school students from a large rural and urban school district in West Virginia participated in 
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this study. The student population came from eight schools and consists of 89% white, 7% 
African-American, 1% Asian and others. We randomly assigned 22 classes (481 students) to the 
explanatory hint condition, 16 classes (282 students) to the non-explanatory hint condition, and 
23 classes (429 students) to the control condition, resulting in a total of 61 classes (1192 
students, 28 teachers participating). All students in a class were in the same condition to prevent 
contamination between students. Also, teachers who had multiple classes were assigned to 
multiple conditions. 

Intervention / Program / Practice: 

The Cognitive Tutor provides students with a sequence of topics (“units”). Students progress 
through the units at their own pace, dependent upon the time needed to master the material. 
Students are judged to have mastered the material if they are able to consistently, correctly 
complete components of the problem and their associated skills, without first making errors or 
asking for a hint (Corbett & Anderson, 1995). In this study, the experimental units covered linear 
equation solving and constituted 5 non-contiguous units in the 40-unit Algebra sequence. 
Students started working through the tutor in sequence with the curriculum from Fall of 2012, so 
the study lasted for the entire 2012/13 school year. All students worked with the same standard 
Carnegie tutor (2012 release) for most of the units except for the experimental units. These 
experimental units included “One-step linear equations”, “Two-step linear equations”, “Linear 
equations with similar terms”, and “Linear equations and the distributive property”. 

Research Design: 

This study contrasted two modified tutors with the standard tutor as a control condition. (Please 
insert Table 1 here) Table 1 summarizes changes included in the two modified versions of the 
tutor and corresponding features of the standard tutor. (Please insert Figure 1 here) The 
experimental versions of the tutor consisted of explanatory and non-explanatory conditions, 
depending on the type of hint messages provided to students. In the explanatory condition, 
students were given direct instruction about how to perform a certain step along with explanation 
on why that problem could be solved in that way. In the non-explanatory condition, students 
were left to discover a problem solution step while a hint message simply gave a direction on the 
tutor interface. For instance, for an equation of -9 + 2w = 3, the explanatory hint was “To 
eliminate -9, add 9 to both sides of the equation because -9 + 9 = 0. In the transform menu, select 
“add to both sides” and enter 9” whereas the non-explanatory hint was “Select an item from the 
transform menu and enter a number.” These hint messages were available on students’ request 
and when students failed to perfonn a correct step in a minute. In the case of wrong attempts, 
identical text prompt appeared in both conditions to instruct students to undo the step and ask for 
a hint. Figure 1 shows an example of the problem-solving environment in the modified Algebra 
Tutor. In both conditions of the modified tutor, largely four major changes were implemented to 
the standard tutor. First, students were given a side-by-side worked example so that they can 
make analogies from example to problem. These examples were randomly selected from a 
predefined pool of problems to prevent superficial analogy that is sometimes observed when an 
example is too similar to a problem. The side-by-side worked example faded when the student 
reached 75% average skill mastery for that unit. Second, several non-verbal scaffolding devices 
were included to highlight problem structure that is not often obvious to students. These include 
a color-coding of variable side of the equation and alignment of equations on the equal (=) signs. 
Students often show difficulty understanding the relation between the transformed equations. 
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Visual alignment was included to help students see these relations. Third, we prevented off-path 
activity by asking students to undo their just performed step when the performed step is 
incorrect. Also, when students fail to perform a correct step for 1 minute, we took it as an 
evidence of floundering and the tutor automatically provided a hint message that corresponds to 
the step without requesting a hint. Fourth, a hint was provided in a single message rather than in 
a progressive manner. The standard tutor uses multiple levels of hints, with each giving 
progressively more specific advice. However, this kind of progressive hint sometimes causes 
poor help seeking behavior. For example, students frequently use bottom-out hints to obtain 
answers without reading prior hint messages that explain the underlying principle of the answer 
(Baker et ah, 2004). 

Data Collection and Analysis: 

At time of writing this report, we had a total of 400 students who completed the first 13 units in 
the Algebra curriculum. We included these completed students (122 explanatory, 103 non- 
explanatory, and 175 control condition) for data analysis. To examine the effect of instructional 
conditions, we focused on the three measures. These include the number of hints per problem, 
the number of errors per problem, and the proportion of mastered skills per section. Among the 
13 units, only units 4, 5, 9 and 13 differed between the experimental conditions. Thus, we treated 
the first three units (1-3) as pre-units, 4-5 units as early experimental units, 6-8 units as early 
non-experimental units, 9 and 13 units as later experimental units, and 10-12 units as later non- 
experimental units. Initial data screening confirmed that there was no difference among the three 
conditions on any of the three measures in the pre-units. A one-way analysis of covariance was 
conducted to determine the effect of instructional conditions on learning when controlling for the 
performance of the pre-units. 

Findings / Results: 

(Please insert Figure 2 here) Figure 2 shows (a) adjusted mean number of hints per problem, (b) 
adjusted mean number of errors per problem, and (c) adjusted mean proportion of mastered skills 
for the three conditions on the early experimental units, early non-experimental units, later 
experimental units, and later non-experimental units. Regarding hints, there was a significant 
effect of instructional conditions in the early experimental units, F{ 2, 396) = 17.58,/? < .001, tj,, 2 
= 082, the early non-experimental units, F{ 2, 396) = 4.43,/? = .012, rj p 2 = 022, and the later 
experimental units, F{ 2, 396) = 26.05,/? < .001, rj p 2 = .116. Students from the both modified 
tutors used more hints than the control condition in both early experimental units and later 
experimental units, ps < .001, but there was no difference between the explanatory and non- 
explanatory condition, /?s > .50. Regarding errors, there was not a reliable mean difference 
among the three conditions in any of the units. Regarding the proportion of mastered skills, there 
was reliable effects of instruction in the early experimental units, F{ 2, 396) = 3.29,/? = .038, rj p 
= .016, and in the early non-experimental units, F{ 2, 396) = 4.30,/? = .014, rj p 2 =.021, but not in 
the later experimental and non-experimental units ( Fs < 1). These effects were because 
explanatory students mastered fewer skills than the other groups of students. In the early 
experimental units, the explanatory students (M= .96, SD = 0.05) mastered fewer skills than the 
non-explanatory students (M= .98, SD = 0.05),/? = .034. In the early non-experimental units, the 
explanatory students (M= .89, SD = 0.07) mastered fewer skills than the control condition (M= 
.91, SD = 0.07),/? = .011. 
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Conclusions: 

To summarize the results, we did not find any benefits of adding verbal explanations. The 
explanatory and non-explanatory students did not differ in terms of number of hints or errors in 
any of the experimental and non-experimental units. In fact, the non-explanatory students 
showed higher proportion of mastered skills than the explanatory students in the early 
experimental units (Figure 2c). The results suggest that students were able to learn without 
verbal direction in certain instructional environments. We attribute the success of the non- 
explanatory group to two main factors. First, a side-by-side example was available while 
students were working on a problem and this helped students to generate a correct problem 
solution step on their own. Second, prevention of off-path activity seemed to reduce unnecessary 
floundering and guide students to correct solution paths. Correctly solved problems become an 
example for later problems. However, when students show excessive floundering, they may not 
be able to remember how they solved a problem (Lewis & Anderson, 1985). 

Regarding the contrast between the modified and standard tutors, the modified tutor 
group requested more hints than the control group in both the early and later experimental units 
(Figure 2a). We believe this greater hint-seeking behavior in the modified tutor is a consequence 
of blocking off-path attempts. In case of wrong attempts, the modified tutor (both explanatory 
and non-explanatory) asked students to undo the just performed step and use a hint. This seemed 
to encourage students to request help whenever they make an error and in turn increase the help 
seeking behavior. In the standard tutor, students can continue after performing a non-optimal 
(but mathematically acceptable) transformation to the equation. More interestingly, there were 
small but significant downstream consequences of the modified tutor on the early non- 
experimental units (but not on the later non-experimental units). As shown in the proportion of 
mastered skills (Figure 2c), the students who used the modified tutor in the experimental units 
mastered fewer skills than those who used the standard tutor in the experimental units although 
both groups used an identical standard tutor in the non-experimental units. Also, the modified 
tutor group requested more hints than the control group. 

There are a number of possible explanations for these downstream effects. First, students 
with the modified tutor may have faced problems re-adapting to the standard tutor when they 
came to the non-experimental units. In the non-experimental units, they were not able to take 
advantage of various scaffolding features (e.g., side-by-side example) that were available in the 
experimental units. Due to the lack of a side-by-side example, they might have requested more 
hints, and in turn gained less learning outcome. Second, reliance on hints on the experimental 
units might have had a persistent effect on the following non-experimental units among the 
modified tutor students. If they learned that their hint requests helped them move forward faster, 
their help seeking behavior might have been reinforced and persisted in later units as well. Third, 
removing off-path activity on the experimental units might have decreased freedom for 
exploration and in turn decreased the attitude of “I can do it on my own.” The modified tutor did 
not allow off-path activity; thus, students did not have an opportunity to observe what happened 
to the equation when a wrong transfonnation was performed. This feature may have decreased 
the feeling of self-control and increased the attitude of relying on the direction. 

In this research we tried to enable students to leam without explicit explanations. We 
realized this objective by providing worked examples and preventing off-path activity to reduce 
unnecessary floundering. Explanatory verbal explanations appear to be not necessary when 
students can learn from examples. 
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Table 1. Summary of characteristics in the three versions of the Carnegie Tutor Algebra. 



Explanatory 

condition 

Non-explanatory 

condition 

Control condition 
(standard tutor) 

Hint - Content 

“how” and “why” 
information 

Infonnation on tutor 
interface 

“how” and “why” 
information 

Hint - Presentation 

Non-progressive single message 

Progressive 

Hint - Voluntariness 

Auto-appearance after 1 minute delay 

None 

Side-by-side 
worked example 

Presents an example next to the problem 
Disappears when average skill mastery = 75% 

None 

Other non-verbal 
scaffolding features 

- Color-coding 

- Alignment between equations 

- Highlighting changed parts between 
transformations 

None 

Error 

Prevents off-path activity, requires an UNDO for 
the incorrectly perfonned step. 

Error flagging 
using an orange 
color 
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Figure 1. An example of a problem-solving environment of the modified tutor. Side-by-side 
worked example is presented on the left side of the screen while students are working on a 
problem. The variable side of the equation is color-coded (green background) and equations are 
aligned with the equal (“=”) sign. When students make an error, the tutor requires them to undo 
their performed step. 
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Figure 2. (a) adjusted mean number of hints per problem, (b) adjusted mean number of errors per 
problem, and (c) adjusted mean proportion of mastered skills for the three conditions on the early 
experimental units, the early non-experimental units, the later experimental units and the later 
non-experimental units. 
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